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METHOD AND APPARATUS FOR 
IMAGE INTERPOLATION BASED ON ADAPTIVE POLYPHASE FILTERS 



By 

5 XIANGLIN WANG 

And 

YEONG-TAEG KIM 

Field of the Invention 

10 The present invention relates to image interpolation, 

an in particular to image interpolations using polyphase 
filters . 



Background of the Invention 

15 Image interpolation is widely used in applications such 

as video deinterlacing, video scaling, etc. in digital TV 
systems. An effective image interpolation technique is 
important for the overall image quality in a digital TV 
system. There are image interpolation techniques that 

20 generate a sharp image, but they give rise to ringing 

artifacts around sharp edges. And, existing techniques that 
generate a smooth image without ringing artifacts, tend to 
smooth the sharpness of edges and make the edges blurring. 
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In order to solve those problems, some adaptive image 
interpolation techniques have been proposed, such as in U.S. 
Patent No. 4,468,693, that provide an interpolation method 
based on a manually operated switch for switching between 
5 different interpolation algorithms depending on the type of 
image being interpolated. However, such a manual operation 
is not suitable for a digital TV system. 



In another example, U.S. Patent No. 5,008,752 provides 
10 several interpolation kernels including cubic convolution, 
bilinear and nearest neighbor interpolation that are 
selected by a control unit based on the difference between 
two neighboring image pixel values. Such a control unit can 
be sensitive to image noise. In addition, bilinear and 
15 nearest neighbor interpolation methods generally produce too 
smooth interpolated images. 



In another example, U.S. Patent No. 5,327,257 provides a 
technique whereby an image is interpolated by combining a 
20 sharp result and a smooth result based on a scaling factor 
with a value from 0 to 1. In this method, 2D filters are 
used to obtain a sharp result and a smooth result from the 
original image. The scaling factor is calculated based on 
lookup tables according to local image contrast and image 
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density. For a given application, this method requires that 
the contrast and density lookup tables be predetermined and 
stored in system memory. 

5 Yet in another example, U.S. Patent No. 6,535,651 

provides a method of interpolating images depending on 
whether a new image pixel belongs to an edge area or a flat 
image area. In order to accurately check if a pixel belongs 
to an edge area as well as the direction of the edge, a 
10 relatively complicated mechanism is required. 

Polyphase filters have been widely used for image 
interpolation purposes due to their computation simplicity 
and good performance. Generally, a polyphase filter used for 
15 digital signal interpolation comprises a low pass filter 
having a cutoff frequency determined by the interpolation 
ratio. For example, to expand a digital signal L times, a 

low pass filter with a cutoff frequency of J/£ can be used. 

The referred frequency is normalized where the normalized 
20 frequency value of 1 corresponds to half sampling frequency. 
The low pass filter used in this case can be separated into 
L sub-filters, with each sub-filter used for a different 
phase of interpolation. More detailed explanation of the 
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polyphase filter concept can be found in "Multirate Systems 
and Filter Banks", P. P. Vaidyanathan, Prentice Hall, 1993, 
incorporated herein by reference. 

5 However, such existing image interpolation techniques 

based on polyphase filters have similar problems mentioned 
above. This is because when the filter is designed to have 
a sharp frequency transition band, it provides a sharp 
interpolated image. But it also generates ringing 
10 artifacts around sharp edges. FIG. 1A shows an example 
frequency response 10 of a low pass filter with a sharp 
frequency transition band. 

When the filter is designed to have a smooth frequency 
15 transition band, it provides a smooth interpolated image 
without ringing artifacts. ' However, the filter tends to 
smooth and blur the sharpness of edges. FIG. IB shows an 
example frequency response 20 of a low pass filter with a 
smooth frequency transition band. The filters in FIGS. 1A- 
20 B, are both 8 tap 4 phase polyphase filters with a cutoff 

frequency of J/£ . When they are used for image 

interpolation, they generate the undesirable results as 
mentioned above. 
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Brief Summary of the Invention 

The present invention addresses the above problems. 
Accordingly, in one embodiment the present invention 
5 provides a method and system for reproducing an original 

image as an interpolated output image in which the original 
image is represented by digital input pixel data. Two 
polyphase filters are used to interpolate a selected image 
position, and the output of the two filters are selectively 
10 combined to achieve a better image interpolation result . 
A controller is designed based on local image high- 
frequency levels and used to adaptively select or combine 
the filter output values, so that the image interpolation 
artifacts mentioned above can be avoided. 

15 

The two filters are designed as low pass filters with 
a cutoff frequency that is directly related to the number 
of phases of the filter. According to said embodiment of 
the present invention, the polyphase filters comprise one- 
20 dimensional (ID) finite impulse response (FIR) digital 

filters. Using a one-dimensional filter, a two dimensional 
image interpolation can be achieved by interpolation along 
horizontal direction and vertical direction separately. 
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In one example, one of the filters comprises a "sharp 
filter" having a sharp frequency transition band, and the 
other filter comprises a "smooth filter" having a smooth 
frequency transition band. The sharp filter and the smooth 
5 filter have different characteristics for image 

interpolation. The output from the sharp filter preserves 
the sharpness of the original image edge. However, it 
generates some ringing artifacts in smooth image areas that 
have a sharp edge in the vicinity. The output from the 

10 smooth filter has no ringing artifact, but originally sharp 
edges tend to be smoothed and blurred. Therefore, 
according to the present invention, the functions of the 
two filters are adaptively (selectively) combined to avoid 
those artifacts. As such, for a given interpolation 

15 position in a video image frame, both the sharp filter and 
the smooth filter are applied. Then the filtering output 
values from the two filters are adaptively combined to 
obtain an interpolation value for the given pixel location. 

20 To adaptively combine the two filter outputs, 

weighting coefficients are calculated for each of the 
filter outputs, based on an estimation of local image high 
frequency level. In one estimation example, for a given 
interpolation position, the neighboring original image 
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pixels at locations/positions that are within the filtering 
range of interpolation are checked. The image high 
frequency is calculated at these original pixel locations, 
and used to estimate the image high frequency level at the 
5 given interpolation position. Then, the weighting 
coefficients are determined based on the estimation 
results . 

As such, if image high frequency level at the given 
10 interpolation position is estimated high, the output from 
the sharp filter is given more weight in determining the 
interpolation value for the given location. Otherwise, the 
output from the smooth filter is given more weight. 
Through such an example adaptive combination of the filter 
15 output values from a sharp filter and a smooth filter in 

the interpolation process, the sharpness of image edges can 
be well preserved without introducing either blurring edge 
artifacts or ringing artifacts. 

20 Other objects, features and advantages of the present 

invention will be apparent from the following specif ication 
taken in conjunction with the following drawings. 
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Brief Description of the Drawings 

FIG. 1A is an example frequency response of a sharp 
filter that can be used for image interpolation; 

FIG. IB is an example frequency response of a smooth 
5 filter that can be used for image interpolation; 

FIG. 2 is a functional block diagram for an image 
interpolation system according to an embodiment of the 
present invention; 

FIG. 3 shows an example interpolation position and the 
10 neighboring original image samples in an image line (or 
column) within an image frame, that can be used for the 
interpolation at the given position; 

FIG. 4 shows a flowchart of an example image 
interpolation method according to the present invention, 
15 wherein image high frequency level at a given position is 
estimated based on the image high frequency components' 
calculated at the two closest original image sample 
positions; and 

FIG. 5 shows a flowchart of another example image 
20 interpolation method according to the present invention, 
wherein image high frequency level at a given position is 
estimated based on the image high frequency components 
calculated at all the original image sample positions that 
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are within the filtering range of interpolation to the 
given position. 



Detailed Description of the Invention 

5 In one embodiment the present invention provides an 

image interpolation method and apparatus that utilizes 
polyphase filters and adaptively selects and/or combines 
their outputs to achieve a better image interpolation 
result. As noted, a control strategy is provided to 
10 adaptively select or combine the filter output values based 
on local image high-frequency levels, so that the image 
interpolation artifacts mentioned above can be avoided. 

According to polyphase filter theory for image 
15 interpolation, the filters are designed as low pass filters 
with a cutoff frequency that is directly related to the 
number of phases of the filter. According to said 
embodiment of the present invention, the polyphase filters 
comprise one-dimensional (ID) finite impulse response (FIR) 
20 digital filters. Using a one-dimensional filter, a two 
dimensional image interpolation can be achieved by 
interpolation along horizontal direction and vertical 
direction separately. 
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As such, an example image interpolation method 
according to the present invention utilizes two polyphase 
filters. One of the filters comprises a "sharp filter" 
having a sharp frequency transition band, and the other 
filter comprises a "smooth filter" having a smooth 
frequency transition band. The sharp filter and the smooth 
filter have different characteristics for image 
interpolation. The output from the sharp filter preserves 
the sharpness of the original image edge. However, it 
generates some ringing artifacts in smooth image areas that 
have a sharp edge in the vicinity. The output from the 
smooth filter has no ringing artifact, but it can smooth 
and blur the sharpness of edges. Therefore, according to 
the present invention, the functions of the two filters are 
adaptively (selectively) combined to avoid those artifacts. 
As such, for a given interpolation position in a video 
image frame, both the sharp filter and the smooth filter 
are applied. Then the filtering output values from the two 
filters are adaptively combined to obtain an interpolation 
value for the given pixel location. 

To adaptively combine the two filter outputs, 
weighting coefficients are calculated for each of the 
filter outputs, based on an estimation of local image high 
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frequency level. In one estimation example, for a given 
interpolation position, the neighboring original image 
pixels at locations/positions that are within the filtering 
range of interpolation are checked. The image high 
5 frequency is calculated at these original pixel locations, 
and used to estimate the image high frequency level at the 
given interpolation position. Then, the weighting 
coefficients are determined based on the estimation 
results . 

10 

As such, if image high frequency level at the given 
interpolation position is estimated high, the output from 
the sharp filter is given more weight in determining the 
interpolation value for the given location. Otherwise, the 

15 output from the smooth filter is given more weight. 

Through such an example adaptive combination of the filter 
output values from a sharp filter and a smooth filter in 
the interpolation process, the sharpness of image edges can 
be well preserved without introducing either noticeable 

20 blurring edge artifacts or ringing artifacts. 

FIG. 2 shows a functional block diagram of an example 
interpolation system 100 that implements the above method 
according to an embodiment of the present invention. The 
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system 100 comprises a sharp polyphase filter 110 (denoted 
as /), a smooth polyphase filter 120 (denoted as g) and a 
control unit 130. Both the sharp polyphase filter 110 and 
the smooth polyphase filter 120 are ID low pass FIR digital 
5 filter. The sharp filter 110 has a sharp frequency 
transition band and the smooth filter has a smooth 
frequency transition band. The sharp filter 110 provides 
sharp interpolation results with good edge quality, but it 
also causes some ringing artifacts in smooth image areas 

10 that have a sharp edge in the vicinity. The smooth filter 
120 does not cause ringing artifacts, however, it may 
smooth and blur the sharpness of edges. The control unit 
130 generates said weighting coefficients that are applied 
to the output from each of the polyphase filters 110, 120, 

15 using combiners (e.g., multipliers) 140 and 150, 

respectively, to generate weighted filter output values. 
Then, the two weighted filter output values are combined 
(e.g., added) together using a summing node 160 to provide 
the interpolation value for a given position. 

20 

According to the example described herein, the sharp 
filter 110 and the smooth filter 120 are ID, FIR, polyphase 
filters. Each of the filters 110 and 120 comprises a group 
of sub-filters, wherein each sub-filter is used for a 
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different phase of interpolation. Assuming each of the 
filters 110 and 120 is a N tap, M phase polyphase filter, 
then each of the filters 110 and 120 has a filter length of 
L = N*M , which can be sub-divided into M sub-filters. For 
5 example, the sub-filters of the filter 110 are denoted as 
f J 9 j = 0,1,--M-1 , wherein j is the interpolation phase. 
Likewise, the sub-filters of the filter 120 are denoted as 
g\ 7 = 0,1, - M-l. The filter length of each sub-filter f j or 
g J is N . 

10 

The value of N can be an odd or even integer. When N 
is an even number, sub-filter coefficients can be denoted 

N N 

as f/ and g/ , where i = -— + l,-",0,---,— . When N is an odd 

number, sub-filter coefficients can be denoted as f/ and 

15 g/ , wherein / = ? --' ? 0 ? --- ? . For simplicity of 

2 2 

explanation in the example described herein, the value of N 
is assumed to be an even number. 



For image interpolation with a fixed and integer 
20 ratio, the value of M can be simply set equal to the 

interpolation ratio. However, for an image interpolation 
application that requires arbitrary or variable 
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interpolation ratios, M should be designed large enough to 
provide a proper interpolation resolution. The number of 
phases of a polyphase filter determines the number of 
essentially different interpolation values that can be 
generated between two neighboring original image pixels. 
With a phase value of M, a polyphase filter can 
interpolate M-l essentially different interpolation values 
between each two neighboring original image pixels. 
Therefore, with a larger value of M , the filter can 
provide better interpolation resolution. In one example, a 
phase value larger than 10 is used for digital TV related 
applications. With a good interpolation resolution, a given 
interpolation position can be approximated by an 
interpolation phase that is closest to it. 

According to the example described herein, it can be 
assumed that the image pixel data input to the system 100 
is a one dimensional data stream. As such, image 
interpolation is conducted along horizontal direction and 
vertical direction separately. If the length of the sub- 
filters is N (N is assumed to be an even number), for a 
given interpolation position, the original' pixel samples 
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within the filtering range of interpolation can be denoted 

• N 1 A N 

FIG. 3 shows such one dimensional image data to be 
5 interpolated. It can be either part of an image line or 
column 300, comprising pixels 310 (solid circles) . FIG. 3 
depicts an interpolation position q 320 (hollow circle) and 
the neighboring original image pixels 310 in an image line 
or column 300, that can be used for the interpolation at 

10 the given position. For the example in FIG. 3, the value N 
is assumed to be 6 for the given interpolation position 
320. The two closest neighboring original pixels 310 for 
the given interpolation position 320 are p 0 and p x . The 
six pixels within the filtering range for the given 

15 position 320 are p i9 i = -2,-1,0,1,2,3 . For description 

simplicity, /?, is used to refer both a pixel's location and 
value in the following. Similarly, q is used to refer both 
an interpolation position and the interpolation value at 
that position. 

20 

Assuming the interpolation phase for the given 
position q is j\(0<j<M), the output values r and s from 
the sharp filter 110 and the smooth filter 120 (FIG. 2), 
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respectively for the position q f can be expressed according 
to relations (1) and (2) below: 

N 

r= £(/.!,*/>,) (D 
s= i,{g J - M *P,) (2) 

2 

5 In relations (1) and (2), filter coefficients of both 

f j and g J are assumed to be normalized (i.e., the sum of 
filter coefficients of either f j or g j is equal to 1) . 

Referring back to FIG. 2, the weighting coefficients 
10 generated by the control unit 130 are combined with the 
filter output values r and s . The final interpolation 
value q of the interpolation position 320 (FIG. 3) is 
performed according to the example relation (3) below: 
q = r*a + s*(\-a) (3) 

15 

wherein a and (\-a) are the weighting coefficients 
generated by the control unit 130 for the sharp filter 110 
and the smooth filter 120, respectively (0<a<l). 
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In relation (3) above, when a takes a value of 1, the 
interpolation value q will be equal to the sharp filter 
output r, which may contain ringing artifacts around sharp 
image edge area. When a takes a value of 0, the 
interpolation value q will have the same value as the 
smooth filter output s, which may have smoothed or blurred 
edge artifacts. According to the present invention, the 
values r and s are adaptively combined to essentially 
eliminate those artifacts, based on the value of a 
calculated by the control unit 130. 

In this example implementation, determination of the 
value of a in the control unit 130 is based on an 
estimation of image high frequency level at the given 
interpolation position 320. The estimation can be obtained 
through the image high frequency components calculated at 
original pixel locations 310 neighboring the interpolation 
position 320. In one case, only the two closest original 
pixel locations to the given interpolation position 320 are 
considered. In another case, all the original pixel 
locations that are within the filtering range of 
interpolation to the given interpolation position can be 
considered. In general, other number of neighboring pixel 
positions on the current image line (or column) , and even 
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some pixel positions from neighboring image lines (or 
columns) may be used. 

As such, in FIG. 3, at an original pixel location p., 
the image high frequency level <j> can be measured through a 
high-pass filtering process (e.g., using a high-pass FIR 
filter) . For example, image high frequency level <f> can be 
simply measured according to the relation: 



wherein the high pass filter is a 3-tap FIR filter 
with coefficients of {-Is, 



In a simple method, image high frequency level at the 
given interpolation position 320 is estimated based on the 
high frequency components calculated at the two closest 
original pixel locations to the given interpolation 
position. As shown in FIG. 3, when the length N of a sub- 
filter is an even number value, the two closest original 
pixels to the interpolation position 320 are p 0 and p x . 
Assuming that the distances between the given interpolation 
position and its two closest original pixels are d 0 and d } 

respectively, and that the distance between neighboring 
original samples is 1, such that: 



4=| A-0.5*(A-,+A + i)l 



(4) 
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«/o+«/,=l (5) 

then, image high frequency components <f> Q and <p x at 
pixel location p 0 and p x , respectively, can be calculated 
5 according to relation (4) above. Then, based on the values 
of (f>Q , <f> x and d 0 , d x , an estimation of the image high 
frequency level, <p , can be further calculated according to 
the relation: 

<p = d { *<f> 0 +d 0 *<f> x (6) 

10 

wherein <p is used as an estimation of the image high 
frequency level at the given interpolation position 320. 

In another embodiment, the image high frequency level 
15 at the given interpolation position 320 can be estimated 
based on the high frequency components calculated at all 
the original pixel locations, within the filtering range of 
interpolation to the given position. In the example shown 
in FIG. 3, when the polyphase filter includes 6 taps, the 
20 pixels p f9 i = -2,-1,0,1,2,3 are within the filtering range to 
the interpolation position 320. In general, when the 
polyphase filter has N taps {N is an even number), then 
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pixels Pi,i = -— + 1," -,0,",— > are all th e pixels within the 

filtering range. Accordingly, image high frequency 

component / = - — + l 5 ---,0,---,— , can be calculated using 

relation (4) above. Then the value of can be calculated 
5 according to relation: 



wherein j is the interpolation phase for the given 
interpolation position and 0<j<M, and M is the number of 
phases of the polyphase filters 110 and 120. 



The value of <p estimated through either relation (6) 
or (7), can reflect the image high frequency level around 
the given interpolation position. If (p has a relatively 
large value, then there is rich image high frequency around 

15 the interpolation position 320. In this case, the output 
from the sharp filter 110 should be given more weight in 
calculating the interpolation value for the given position 
so that edges can be better interpolated without being 
smoothed or blurred. On the other hand, if (p has a 

20 relatively small value, it means that the given 

interpolation position 320 is in a flat image area. In 




(7) 
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this case, the output from the smooth filter 120 should be 
given more weight in calculating the interpolation value so 
that ringing artifacts can be avoided. 

5 Once the estimation of the image high frequency level 

q> around the given interpolation position is available, the 
weighting coefficient a can be calculated according to the 
relation: 

or = min(l, maxCO^-r,)/^-^))) (8) 

10 

wherein T x and T 2 are two pre-determined threshold 
values (T 2 >T x >0) . According to relation (8), when the value 
of (p is greater than T 2 , then a takes a value of 1. In 
this case, the output from the sharp filter 110 is used as 

15 the interpolation value for the given interpolation 

position 320. When the value of <p is smaller than T x , then 
a takes a value of 0. In this case, the output from the 
smooth filter 120 is used as the interpolation value for 
the given position. And, when the value of <p is between T x 

20 and T 2 , then a takes a value between 0 and 1. In this 

case, the output values from the sharp filter 110 and the 
smooth filter 120 are mixed together as the interpolation 
value q for the given position 320. 
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In one implementation, the values of 7 1 , and T 2 are 
determined in an empirical fashion. To do so, once the 
sharp filter 110 and the smooth filter 120 are designed, a 
5 group of testing images can be selected. Image 

interpolation is performed with different interpolation 
ratios with these test images accordingly. During this 
process, the values of T x and T 2 can be manually adjusted 
according to the interpolated result. If obvious ringing. 
10 artifacts appear, the values of T x and T 2 can be adjusted 

higher. Otherwise, if there is blurred edge artifact due to 
interpolation, the values of 7j and T 2 can be adjusted 
lower. 

15 FIG. 4 shows an example flowchart of the image 

interpolation method above, wherein image high frequency 
level at a given position is estimated based on the image 
high frequency components calculated at the two original 
image sample positions that are closest to the given 

20 position. As such, initially image data 
N N 

Pi,i = — — + l 9 - , - 5 0,---,— , is obtained, and the interpolation 

phase j is determined (step 400) . Then outputs r and s of 
the sharp and smooth filters 110 and 120, respectively, are 
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determined (step 410). Using the above values $ 0 and <j> x , 
and d Q and d } , image high frequency level <p at the given 
interpolation position is determined as <p = d x + d 0 * ^ (step 
420) . The weighting coefficient is determined as 
5 a = min(l, max(0, (<p-T x )/(T 2 -7]))) (step 430), and the interpolation 
value is calculated as q = r*a + s*(\-a) (step 440) . It is 
determined if all necessary positions have been 
interpolated (step 450) . If so, the process ends, 
otherwise the next interpolation position is selected for 
10 processing (step 460) and the process returns to step 400. 

FIG. 5 shows the flowchart of another embodiment of 
image interpolation according to the present invention, 
wherein image high frequency level at a given position is 
15 estimated based on the image high frequency components 

calculated at all the original image sample positions that 
are within the filtering range of interpolation to the 
given position. 

N N 

20 As such, initially image data p i9 i = + (),•••, — , is 

2 2 

obtained, and the interpolation phase j is determined (step 
500) . Then outputs r and s of the sharp and smooth filters 
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110 and 120, respectively, are determined (step 510) . 
Using relation (7) above, the image high frequency level <p 
at the given interpolation position is estimated based on 
N N 

values $ g9 i = — — + l,---,0,---,— (step 520). The weighting 

5 coefficient is determined as a = min(l, max(0,(#>-7i)/(r 2 -7]))) 
(step 530), and the interpolation value is calculated as 
q = r*a + s (step 540) . It is determined if all 

necessary positions have been interpolated (step 550) . If 
so, the process ends, otherwise the next interpolation 
10 position is selected for processing (step 560) and the 

process returns to step 500. As those skilled in the art 
will recognize, the process steps in Figs. 4 and 5 are 
performed for the vertical and horizontal direction 
separately according to the relations detailed above. 

15 

Using the above example method, an image can be 
interpolated in an adaptive manner with polyphase filters. 
Both the blurred edge artifact and the ringing artifact can 
be successfully avoided in the interpolated image. In 
20 addition, because the polyphase filters are ID FIR filters, 
computation requirement and complexity of a system 
according to the present invention is low, making such a 
system suitable for real-time applications. Although in 
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the example description herein two polyphase filters are 
utilized, as those skilled in the art will recognize, the 
present invention contemplates using three or more 
polyphase filters having different frequency transition 
bands, ranging from very smooth to very sharp. In that 
case the controller 130 determines weighting coefficients 
for each of the plurality of the filters. Then the 
weighted output of the filters is combined to determine the 
interpolated output value. 

While this invention is susceptible of embodiments in 
many different forms, there are shown in the drawings and 
will herein be described in detail, preferred embodiments of 
the invention with the understanding that the present 
disclosure is to be considered as an exemplification of the 
principles of the invention and is not intended to limit the 
broad aspects of the invention to the embodiments 
illustrated. The aforementioned system 100 according to the 
present invention can be implemented in many ways, such as 
program instructions for execution by a processor, as logic 
circuits, as ASIC, as firmware, etc., as is known to those 
skilled in the art. Therefore, the present invention is not 
limited to the example embodiments described herein. 
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The present invention has been described in 
considerable detail with reference to certain preferred 
versions thereof; however, other versions are possible. 
5 Therefore, the spirit and scope of the appended claims 

should not be limited to the description of the preferred 
versions contained herein. 
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